Skip to content

support for BLOB - #152

Open
julesyan wants to merge 11 commits into
mainfrom
blob-support
Open

support for BLOB#152
julesyan wants to merge 11 commits into
mainfrom
blob-support

Conversation

@julesyan

@julesyan julesyan commented Jun 3, 2026

Copy link
Copy Markdown
Collaborator

BLOB Support — Streaming via HTTP Token Endpoint

Overview

This PR adds full BLOB/BINARY support to mapepire, covering both reading BLOB columns from query results and writing BLOB values as prepared statement parameters.

The key design decision is how BLOB data is returned in daemon mode (WebSocket server). Rather than inlining potentially large binary payloads as Base64 in the JSON response — which would be bounded by the WebSocket message size limit — the server stores each BLOB value temporarily and returns a short-lived HTTP token URL instead. The client fetches the actual bytes via a separate GET /blob/{token} HTTP request. This means there is no upper bound on the size of a BLOB that can be transferred.

Daemon Mode — BLOB Output Shape

When a query returns a BLOB/BINARY column in daemon mode, the column value in the JSON response is not a string. Instead it is an object:

{
  "JBLOB": {
    "blob_url": "/blob/550e8400-e29b-41d4-a716-446655440000",
    "size": 52428800
  }
}

The client then fetches the raw bytes via:

GET https://<host>:<port>/blob/550e8400-e29b-41d4-a716-446655440000
Authorization: Basic <same credentials used for the WebSocket connection>

The response is application/octet-stream — raw binary bytes, no encoding.

Token properties:

Single-use — consumed and deleted on first successful retrieval
TTL — expires after 60 seconds by default (configurable via BLOB_TOKEN_TTL env var)
Auth-protected — the Authorization header on the GET must match the credentials of the WebSocket connection that produced the token; mismatched credentials return 401 and discard the token
Memory/disk — BLOBs ≤ 1MB are held in heap; larger ones are spooled to a JVM temp file to bound memory pressure

Single Mode — BLOB Output Shape

In --single mode there is no HTTP server, so BLOB values fall back to inline Base64 — the same as the previous behaviour for clients using stdin/stdout.

BLOB Input (Parameters)

BLOB parameters in prepared statements are sent as Base64-encoded strings — the same as any other string parameter. The server detects the JDBC parameter type and automatically decodes the Base64 before calling setBytes() / setBinaryStream(). No special client-side encoding is needed beyond Base64.

I have attached three test files I used, the second and third one requires a table made from:

create table juliayan.tempblob ( jblob blob(100) )

test-blob.txt
test-blob2.txt
test-blob-daemon.js

For testing blob deamon, start the deamon mode on any port, then run the command below on your machine to run

node test-blob-daemon.js <host> <user> <password> <port>

@julesyan
julesyan requested a review from ThePrez June 3, 2026 19:27
This was linked to issues Jun 3, 2026
Comment thread src/main/java/com/github/ibm/mapepire/http/BlobServlet.java Fixed
… through an error message'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
@julesyan

julesyan commented Jun 8, 2026

Copy link
Copy Markdown
Collaborator Author

Before this change, fetching a row with a large BLOB column would block until all the bytes were saved to disk, then send the response. Now the response goes back immediately with a blob_url, and the bytes are saved in the background. The client can fetch the URL whenever it's ready.

@julesyan
julesyan marked this pull request as draft June 8, 2026 17:45
Fixed getOutputParms() length bug and tracking bug for large BLOBs
Cleaned up unreachable code
Fixed Content-Length to use actual size
Sanitize log injection for BlobServlet
@hisham-sid
hisham-sid requested a review from eric-simpson August 5, 2026 14:24
@hisham-sid
hisham-sid marked this pull request as ready for review August 5, 2026 14:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Blob support Improve LOB support

3 participants